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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )g] Responsive to communication(s) filed on 28 April 2003 . 
2a)E3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1-26 is/are pending in the application. 

4a) Of the above claim(s) 23-26 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) IEI Claim(s) 1-22 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) KI Claim(s) 23-26 are subject to restriction and/or election requirement. 
Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) 13 The proposed drawing correction filed on 28 April 2003 is: a)D approved b)^ disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1.0 Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 

3-D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-1 52) 

3) □ Information Disclosure Statement(s) (PTO-1 449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 
Election/Restrictions 

1 . Restriction to one of the following inventions is required under 35 U.S.C. 121 : 

I. Claims 1 -22 5 drawn to a robot system having an image processing function, 
classified in class 382, subclass 153. 

II. Claims 23-26, drawn to a method for automatically determining an arrangement 
of a workpiece relative to a robot, classified in class 382, subclass 154. 

The inventions are distinct, each from the other because of the following reasons: 

2. Inventions I and II are related as combination and subcombination. Inventions in this 
relationship are distinct if it can be shown that (1) the combination as claimed does not require 
the particulars of the subcombination as claimed for patentability, and (2) that the 
subcombination has utility by itself or in other combinations (MPEP § 806.05(c)). In the instant 
case, the combination as claimed does not require the particulars of the subcombination as 
claimed because the robot system of group I does not claim all the limitations set forth in group 
II such as the imaging device being affixed to an arm of the robot, storing images of an 
workpiece/object shaped like the workpiece in the reference image, or capturing a working 
image of the workpiece from a known arrangement of the robot. The subcombination has 
separate utility such as a method for determining an arrangement of a person or other specific 
object (workpiece) relative to an image capturing device, camera, or the like (mounted on a 
robot). 

3. Newly submitted claims 23-26 directed to an invention that is independent or distinct 
from the invention originally claimed for the following reasons: set forth above. 
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Since applicant has received an action on the merits for the originally presented 
invention, this invention has been constructively elected by original presentation for prosecution 
on the merits. Accordingly, claims 23-26 are withdrawn from consideration as being directed to 
a non-elected invention. See 37 CFR 1.142(b) and MPEP § 821.03. 

Drawings 

4. The proposed drawing correction filed on April 28, 2003 has been disapproved because it 
is not in the form of a pen-and-ink sketch showing changes in red ink or with the changes 
otherwise highlighted. This is mainly in regards to the changes to Figure 6, which were not 
suggested and have not been explained, highlighted, or shown in red ink. In addition, the Office 
action mailed January 28, 2003 suggested that figures 4 and 8-10 be labeled as "Prior Art." The 
present amendment has only labeled figures 4, 8, and 10; perhaps figure 9 was erroneously 
omitted. See MPEP § 608.02(v). 

Response to Arguments 

5. Applicant's arguments, see page 1, second to last paragraph, filed April 28, 2003, with 
respect to Figures 4, 8, and 10 have been fully considered and are persuasive. The objection of 
Figures 4, 8, and 10 has been withdrawn. Figure 9, however, should also be labeled as "Prior 
Art" as set forth in the last Office action. The changes made to Figure 6 should be highlighted or 
shown in red ink. 

6. Applicant's arguments at the bottom of page 7, filed April 28, 2003, with respect to 
correction of formalities in the claims have been fully considered and are persuasive. The 
objection of claims 2, 6, 1 1, 13, and 22 has been withdrawn. 
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7. Applicant's arguments filed April 28, 2003 with respect to the 35 USC 1 12, 2 nd paragraph 
rejection has been fully considered but they are not persuasive. As recited below, the 35 USC 

1 12, 2 nd paragraph rejection with respect to claims 8-10 and 19-21 stands since the "second 
image data capturing device" is not mentioned in either of claims 1 or 12. It is noted that 
applicant's version of the claims apparently does not include the preliminary amendment as 
originally filed with the application which amended claims 8, 9, and 10 to depend only from 
claim 1, and claims 19, 20, and 21 to depend only from claim 12. 

8. Applicant's arguments with respect to claims 1 and 12 have been considered but are moot 
in view of the new ground(s) of rejection. The new grounds of rejection is due to the added 
limitations that each reference model is created based on an image of a reference object captured 
by said image capturing device in a different direction, and for each reference model storing 
information of the capturing direction of its associated image, and using said reference models to 
select one of the reference models whose image of the reference object matched with the image 
of the object of detection (emphasis added). 

9. Regarding applicant's arguments on page 8, last full paragraph, applicant states that 
"Suzuki's reference models (part-forms) are not 'based on image data of a reference object,' and 
do not have associated capture directions." The examiner agrees with this assertion but would 
like to point out that Suzuki also discloses that the sample 10 (reference object) is viewed from 
various directions by moving a pair of CCD cameras around the sample, or by rotating the 
sample, and capturing 2-dimensional image data, thereby data of a portion which cannot be seen 
from a single direction can be obtained (column 3, line 66-column 4, line 9). Suzuki then 
discloses that the 3-dimensional positions of the vertices and edges of the sample are detected 
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and the parts that make up the sample are detected and compared to those stored in the part-form 
memory (column 4, line 10-24). Suzuki goes on to disclose that the construction of the sample is 
stored in the construction data memory 21 including position data representing 3 -dimensional 
positions, and posture data representing orientation at the corresponding positions of the parts 
included in the sample (column 4, line 25-39). 

10. Regarding applicant's arguments at the bottom of page 9, applicant states "Suzuki does 
not determine an operation orientation/position based on captured direction and orientation." 
The examiner respectfully disagrees. Suzuki discloses a task planning module 13 which directs 
the robot to perform tasks for moving parts from a conveyor to the positions determined by the 
part arrangement data stored in construction data memory 21 . Motion command data (robot 
operation) is defined from position data for gripping parts on the conveyor, position data of 
destinations of the parts, and orientation data at the corresponding positions (column 4, line 49- 
column 5, line 4). 

1 1 . Regarding applicant's arguments on page 10, first two paragraphs, applicant states 
"Suzuki fails to discuss or suggest performing matching processing on an image of the object of 
detection with reference to models. . ." The examiner respectfully disagrees. As discussed above, 
Suzuki captures images of a sample 10 from various directions and detects the construction 
thereof by comparing the outline data of the sample with form data of each part stored in part- 
form memory 19, thereby specifying parts included in each layer (column 4, line 17-24). Suzuki 
then discloses that the robot performs tasks for moving the parts from the conveyor to the 
positions determined by the part arrangement data stored in construction data memory 21 using 
the position data for gripping parts on the conveyor, position data of destinations of the parts, and 
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orientation data at the corresponding positions. It is also disclosed that the positions where the 
robot grips the respective parts are automatically detected using the CCD camera pair (column 4, 
line 49-column 5 3 line 1 1). 

Claim Rejections - 35 USC § 112 

12. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

13. Claims 8-10 and 19-21 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. Claims 8-10 (amended in pre-amendment filed April 10, 
2000 to depend only from claim 1) and 19-21 (amended in pre-amendment filed April 10, 2000 
to depend only from claim 12) recite the limitation "said second image data capturing device" in 
line 2-3. There is insufficient antecedent basis for this limitation in the claim. 

Claim Rejections - 35 USC §103 

14. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

15. Claims 1-4 and 12-15 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Suzuki (USPN 4,835,450, reference of record) in view of Kelley et al. (USPN 4,305,130, 
reference of record), hereafter Kelley. 

16. Regarding claim 1 , Suzuki discloses a robot system having an image processing function 
for determining orientation, or orientation and position of a robot operation on an object of 
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detection (column 4, line 25-39), the system comprising: a robot (figure 1, element 3); a first 
image capturing device (figure 1, element 1 la); a memory storing reference models (figure 2, 
element 21) each created based on an image of a reference object (sample 10) captured by said 
image capturing device in a different direction (column 3, line 66 to column 4, line 9), and for 
each reference model storing information of the capturing direction of its associated image and 
information of an orientation of the robot with respect to the reference object (column 4, line 32- 
39), said reference object (figure 1, element 10) being the object of detection or having a shape 
substantially identical (figure 1, element 15) to that of the object of detection (column 3, line 57- 
62); and a processor (figure 2, element 18) to perform matching processing on an image of the 
object of detection (captured by said first image capturing device; column 4, line 17-24), and to 
determine the orientation, or the orientation and position of the robot operation to be performed 
on the object of detection, the determining based on the selected image of the reference object, 
based on said one reference model and the information of its associated capturing direction 
(column 4, line 49-column 5, line 4), and based on the information of the orientation of the robot 
with respect to the reference object that is associated with said one reference model (column 5, 
line 5-11). 

17. Suzuki does not explicitly disclose using said reference models to select one of the 
reference models whose image of the reference object matches with the image of the object of 
detection. Kelley discloses an apparatus and method to enable a robot with vision to acquire, 
orient, and transport workpieces wherein workpiece features are matched to stored feature 
models and if there is one or more matches, the model with the best match is selected, which is 
associated with a particular hand- workpiece transformation (orientation) matrix (column 13, line 
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42-58). It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to select one of the reference models whose image of the reference object matches 
with the image of the object of detection as taught by Kelley in order to acquire a workpiece 
from randomly arranged workpieces by examining data which matches holdsites and places the 
acquired workpiece at a goal site without requiring complete knowledge of the position and 
orientation of a workpiece in a bin (column 1, line 35-47). 

18. Regarding claim 12, which is substantially the same as claim 1 except that claim 12 
includes the limitation that the memory stores reference models created based on image data of 
different kinds of reference objects, Suzuki discloses a robot system having an image processing 
function including a memory (construction data memory 21) storing reference models created 
based on image data of different kinds of reference objects captured by said first image capturing 
device (column 4, line 25-39). Additionally, Suzuki discloses that when the design 
(arrangement) of the sample 10 is changed, the new design needs only to be imaged and stored 
so that the product can be constructed (column 6, line 51-62). See discussion of claim 1 above 
for all other common features. 

19. Regarding claims 2 and 13, Suzuki discloses that said reference models are obtained from 
a part of the image data of the reference object (column 4, line 2-9). 

20. Regarding claims 3 and 14, Suzuki discloses that said reference models are obtained by 
processing the image data of the reference object (column 4, line 5-16). 

21 . Regarding claims 4 and 1 5, Suzuki discloses that said first image capturing device 
comprises a camera for capturing two-dimensional image data (column 3, line 66 to column 4, 
line 5). 



// 
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22. Claims 5 and 16 are rejected under 35 U.S.C. 103(a) as being unpatentable over Suzuki in 
view of Kelley as applied to claims 1 and 12 above, and further in view of Suyama et al. (USPN 
4,879,664, reference of record), hereafter Suyama, or Stauffer (USPN 4,410,804, reference of 
record). 

23. Suzuki does not disclose that the image data are captured from a predetermined distance. 
Suyama discloses a three-dimensional position sensor comprising robot-teaching apparatus 
wherein said image data of the reference object are captured by said camera (figure 11a, element 
35) from a predetermined distance (column 6, line 20-37). In addition, Stauffer teaches that if a 
two-dimensional sensor is used, the image processor is unable to simultaneously determine the 
distance to the object unless the objects are always positioned at a known distance (column 1, 
line 3 1-39). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to capture an image at a predetermined distance in order to calibrate a three- 
dimensional sensor as taught by Suyama or such that the distance to the object need not be 
determined by other means as taught by Stauffer. 

24. Claims 6, 7, 1 1, 17, 18, and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Suzuki in view of Kelley as applied to claims 1 and 12 above, and further in view of Maeno 
et al. (USPN 5,047,714, reference of record), hereafter Maeno. 

25. Regarding claims 6 and 17, Suzuki discloses a step of determining the orientation and/or 
position of the object and moving the robot arms to have said orientation and/or position, but 
does not disclose that the robot moves a second image capture device to have the determined 
orientation and/or position. Maeno discloses that said robot system further comprises a second 
image capturing device (figure 7c, element 10) and said robot situates said second image data 
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capturing device to have said determined position (column 4, line 47-49) with respect to the 
object, and said processor processes second image data captured by said second image capturing 
device to detect position and/or posture of the object with respect to said second image data 
capturing device (column 4, line 54-63). It would have been obvious to one of ordinary skill in 
the art at the time the invention was made to move the second image capture device to have the 
determined orientation and/or position as taught by Maeno in order to provide fine-tuning 
adjustments to the orientation and/or position determined by the first image capture device. 

26. Regarding claims 7 and 18, Suzuki discloses that said robot system further comprises a 
second image capturing device (figure 1, element 1 lb) for obtaining three-dimensional position 
(column 4, line 5-16) and that said second image data capturing device is directed to a 
characterizing portion of the object (column 3, line 66 to column 4, line 5), but does not disclose 
that the robot moves the second image capture device to have the determined orientation and/or 
position. Maeno discloses that said robot situates said second image data capturing device to 
have said determined position with respect to the object; and said processor processes second 
image data captured by said second image capturing device to detect position and/or posture of 
the object with respect to said second image data capturing device (see discussion of claims 6 
and 17 above). 

27. Regarding claims 1 1 and 22, Suzuki does not expressly disclose that said robot operation 
is an operation of picking up at least one object from a plurality of objects overlapped with each 
other. Kelley discloses an apparatus and method to enable a robot with vision to acquire, orient 
and transport workpieces including a robot operation of picking up at least one object from a 
plurality of objects overlapped with each other (figure 9; column 13, line 6-15). It would have 
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been obvious to one of ordinary skill in the art at the time the invention was made to pick objects 
from a plurality of overlapping objects as taught by Kelley in order to retrieve randomly oriented 
parts from a bin to save time by automatically orienting the parts for manufacturing or the like 
(column 1, line 12-32). 

28. Claims 8-10 and 19-21 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Suzuki in view of Kelley as applied to claims 1 and 12 above, and further in view of Sakakibara 
et al. (JP 07-270137, reference of record with translation), hereafter Sakakibara. 

29. Regarding claims 8-10 and 19-21, see 35 U.S.C. 1 12 second paragraph rejection above. 

30. Regarding claims 9 and 20, Suzuki does not disclose a second image capturing device 
comprising a three dimensional visual sensor of spot-light scanning type. Sakakibara discloses a 
three dimensional visual sensor usable in robot automation (page 1, paragraph 1), of spot-light 
scanning type capable of measuring distance between the sensor and an object (page 3, 
paragraph 15). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to utilize a three dimensional visual sensor of spot-light scanning type as 
taught by Sakakibara in order to accurately determine the three dimensional position of an object 
using one device. 

3 1 . Regarding claims 10 and 21, Suzuki does not disclose a second image data capturing 
device comprising a structured-light unit for irradiating a structured light on an object. 
Sakakibara discloses an image data capturing device comprising a structured-light unit for 
irradiating a structured light on an object and capturing an image of the object including the 
irradiated light on the object (page 3, paragraph 16). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to utilize a structured light unit as 



Application/Control Number: 09/546,392 Page 12 

Art Unit: 2623 

taught by Sakakibara in order to accurately determine the three dimensional position of an object 
using one device. 

Conclusion 

32. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-872-9314 for regular 
communications and 703-872-9314 for After Final communications. 
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Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-306-0377. 




